Peer Review in Analytical Work: Structured Challenge

Structured challenge is picking questions so that each one could catch something the others cannot, then running them rather than debating them. Six such questions, put to five worked cases, catch 0, 3, 1, 1, 2 and 1 of them. The question every reviewer reaches for first, whether the arithmetic reconciles, catches none of the five.

Each of the five cases was rebuilt from the parts the earlier work defined for it, and each of the six challenges was then run against each rebuilt case as a stated test, so the thirty answers in the grid are outputs rather than opinions.

Reproducibility settles a narrow and useful thing: a second person, handed the trail, reaches the same number. Structured challenge starts one step past reproducibility. The trail is clean, the recomputation agrees, and somebody now has to decide whether the work is any good. Reproduction and quality are different questions, and passing the first says almost nothing about the second.

Three facts about reviewing carry the argument, and none of them needs a new figure. The first is that a review is a search, and a search is only as good as the set of places it looks, so the useful unit is the set of questions rather than any single clever one. The second is the five cases already seen going wrong. Every one of them reconcilesTo recompute a stated figure from the stated data and find that the two agree. Reconciling checks the sum, not whether the sum was worth taking. perfectly, and perfect reconciliation is exactly what makes them useful. The third is that coverage is countable: whether a set of questions reaches every case on a list is something that can be worked out, not something two people have to argue about across a table.

Here are the five, named the same way in every block below so that one of them can be followed all the way through. Case A, the manufactured pattern: a fitted line reading 0.7559 whose misses run in streaks. Case B, the chair count: a count of the chairs somebody set out in an invented hall. At 0.9029 the chair count fits the outcome better than the genuine relationship does. Case C, the fifty month record: a mean of 0.50 per cent drawn from a mechanism whose true mean is 1.00 per cent. Case D, the near duplicate: a second input almost identical to the first. The near duplicate leaves the fit at 0.7560 and drives a coefficient from 1.5000 to minus 1.0000. Case E, the leak: a three month average accidentally centred, so it uses one month that had not happened yet, and improves by 48.1650 per cent for that reason alone.

What is review for in analytical work, and what is it not for?

Start with the three things review is not. A signature is a social act and adds nothing to a figure, so review is not approval. Two opinions about the same evidence are still one body of evidence, so review is not a second opinion on the conclusion. And it is not a repeat of the arithmetic, though that is what most reviews turn into within about four minutes of opening the file.

Review is a structured attempt to find the observation that would make the work wrong. The reviewer's job is to name the observation that would overturn the claim, and then go and look for it. Naming that observation is the same discipline the design was supposed to have before the work started, done a second time by somebody with no stake in the answer. When it is done properly the reviewer is not marking; the reviewer is searching, and the author is the one holding the torch.

The everyday version is a shopkeeper at closing time. She asks a friend to check the day's total, and the friend adds the column again and agrees. The second count was worth a minute. Now imagine she asks the friend a different question: what would make today's total meaningless? The friend, who has no stake in the day, says that the till was moved to the back at four when the delivery came, and that everything sold at the front counter after four went into a cash box nobody has opened. The first question checked the addition. The second one found the money.

Why does the challenge everybody makes first catch nothing?

Open any of the five cases and recompute it. Case A's fit really is 0.7559 and its misses really do read 0.4862 when taken in record order. Case B's chairs really do reach 0.9029. Case C's record really does average 0.50 per cent across its fifty months. Case D's fit really is 0.7560 and the coefficient on its first input really is minus 1.0000. Case E's leaked average really is 48.1650 per cent closer than the honest one. Every figure in every case survives being recomputed from the data it claims to come from.

So the challenge every reviewer makes first catches none of the five, and the count is zero rather than small. The zero is not bad luck in the choice of cases. The zero follows from what the cases are. None of the five was an arithmetic mistake, so a question about arithmetic was never going to reach any of them. A review that consists of checking the sums has checked the sums.

Which questions catch which kinds of mistake?

Six questions, chosen so that each one looks somewhere the others do not. Each is applied as a stated test with a stated thresholdA line fixed in advance, so that what falls on either side of it was decided before anything was read., and the answer is computed rather than felt. Here they are, with the reading each one produces.

1. Does the stated arithmetic reproduce? Recompute every figure from the data it claims to come from. All five say yes.
2. Do the misses, read in record order, sit above a third in size? Take the leftovers from the fitted line, keep them in the order the record ran, and read them against themselves one step back. Case A reads 0.4862, case B reads minus 0.1908, case D reads 0.4822 and case E reads minus 0.5315. Case C has no fitted line to read, so the question does not reach it.
3. Can the mechanism be stated without a number? Say out loud why the input should move the outcome, using no arithmetic at all. Four of the five have an answer. Case B does not. The answer would have to be that chairs put out in a hall drive the outcome.
4. Could every input have been known at the time it is attached to? Take each input and ask on what date it existed. Case E fails: its three month average is centred, so the figure attached to a month includes the month after it.
5. Does the range around the headline figure allow a value that reverses the claim? Case A's slope range runs from 0.8050 to 2.1950 and case B's from 0.6957 to 1.2036, neither of which reaches zero. Case C's range on the mean runs from minus 0.8788 to 1.8788 per cent, and case D's range on its first coefficientThe multiplier attached to one input inside a fitted relationship. It says how much the outcome is credited to that input once the others are held still. runs from minus 94.2860 to 92.2860. Both of those contain zero.
6. How alike are the inputs to each other? Case D's two inputs sit at 0.999967 of each other, with a variance inflation factorA figure saying how much one input being close to another widens the range around its multiplier. It rises without limit as two inputs approach each other. of 15001. No other case has a second input at all.

The counts are 0, 3, 1, 1, 2 and 1, and every one of the thirty answers behind them was computed by running the test rather than by deciding what it would probably find. The distinction between running a test and guessing what it would find is the whole reason to draw the grid. A matrix of opinions laid out like a matrix of results would be precisely the failure this discipline exists to prevent, and it would look identical when printed.

People misread one row of that grid, so it is worth pausing on. The question about the range does not ask whether the headline figure is large. The question asks what else the data would have permitted. On case D that turns out to be almost anything.

How few questions can still reach every case?

Read the grid down its columns rather than across its rows and something sharper appears. Case D is reached three different ways, by the misses, by the range and by the likeness of its inputs, so it is nearly impossible to miss. Case E is reached two ways. But case A is reached only by the question about the misses in record order. Case B is reached only by the question about mechanism. Case C is reached only by the question about the range.

Three of the five cases are caught by exactly one question each, so dropping any one of those three questions lets a whole case walk through the review untouched. Dropping a question and losing a whole case is a much stronger claim than saying the questions are all useful. Those three questions are not substitutes for anything. No amount of care applied to the other five recovers what dropping one of them loses.

Coverage turns into a counting problem. There are six questions, so there are sixty three sets that can be taken from them, counting every set from one question up to all six. Running all sixty three and asking which of them reach all five cases gives an answer: eight of the sixty three do. Of those eight, one has three questions in it, three have four, three have five, and one has six.

The smallest set that reaches every case has three questions in it, and it was found by searching all sixty three combinations rather than by picking the three that looked strongest. The three are the misses in record order, the mechanism said with no number, and what else the range allows. There is exactly one set of three that works, and every one of the eight covering sets contains all three of those questions. No other question can stand in for any of the three. The question about recomputation appears in four of the eight and is spare in every one of them.

What does it mean when a check raises a flag but explains nothing?

Case E is the leak, and it is where a reviewer who trusts diagnosticsComputed checks whose only job is to raise a flag, rather than to produce a figure anybody reports. gets into the most trouble. Put the question about the misses in record order to it. The leaked version's misses read minus 0.5315 at lag oneThe comparison of each reading in a record with the one immediately before it, kept in the order the record ran., comfortably past a third in size, so the test fires and the flag goes up. So far the check looks like it is doing its job.

Now run the same test on the honest version of the same work, the three month average that uses only months that had already happened. Its misses read 0.5740 at lag one, further past the threshold than the leaked version's reading. Both versions fail the same test, so the flag is real and it diagnoses nothing at all. A reviewer who reads the flag as evidence of the leak has read something that would have appeared whether the leak was there or not.

The fourth question is what actually separates them, and it is not a number. Ask on what date each input existed. The honest average, sitting on a month, uses that month and the two before it. The leaked one uses the month before, the month itself and the month after, and the month after had not happened yet when the figure was needed. Dating an input is a question about construction, answered by reading how the average was built, and no reading of the outputs can substitute for it.

Which question can no arithmetic ask?

Case B is the chair count, and it is the one that ends careers quietly. Six arithmetic measures were put side by side against the genuine relationship, and the chairs win all six: a fit of 0.9029 against 0.7559, a correlation of 0.9502, misses left over of 86.7349 against 218.0000, a reading against zero of 8.6236 against 4.9770, misses at lag one of minus 0.1908 against 0.4862, and a split halfFitting the same relationship on each half of a record separately, then putting the two answers side by side to see how far apart they land. agreeing to 0.0014 where the genuine relationship's halves agree only to 0.1893.

A reviewer armed with every diagnostic available and no question about mechanism will sign the chair count off, and the failure sits in the question set rather than in the reviewer. The instinct on reading case B is to assume somebody was sloppy, so the point is worth saying plainly. Nobody was. Every measure that could be computed was computed, and every one of them said the chairs were better.

The six questions can be placed by how many cases each one catches against whether arithmetic can answer it at all. Two of the six carry no arithmetic: the one about mechanism and the one about dating. Between them they are the only route to case B and the only thing that separates case E's two versions. No arithmetic question in the set reaches case B at all. A question set built entirely out of diagnostics therefore has a hole in it that no amount of extra diagnostics can fill.