The Model: A Deliberate Simplification, and What That Costs

A model is a rule that turns inputs into a statement about an outcome, and every model is a decision to ignore things. On the ten paired months of the Nakshatra unit and the Vasant unit the straight line misses by 218 in squared terms, of which 72 is the price of insisting the relationship is straight and 146 is beyond the reach of any rule reading that one input at all.

Two things are already settled, and both are used below exactly as they stand, with no second pass over either. There is a fitted line on ten paired monthly readings, and its sum of squared missesTake the gap between what a rule said and what actually happened in each month, square every gap so that overshooting and undershooting both count as damage, then add the ten results together. is 218. There is an R squaredThe share of the up and down movement in whatever column is being explained that a rule accounts for, read as a number between nought and one. Where it comes from is settled elsewhere in these notes. of 0.7559 attached to that same line. A slope, an R squared, a miss, and what goes wrong when two inputs say the same thing were all settled earlier, so none of them is re-explained.

The readings themselves were made up for teaching. The Nakshatra unit changed by 1.00, 6.00, minus 4.00, 11.00, 1.00, minus 9.00, 6.00, 1.00, minus 4.00 and 1.00 per cent across ten months in that order. The Vasant unit changed by 3.00, 18.50, 2.50, 17.00, minus 3.00, minus 13.00, 6.50, minus 1.00, minus 7.50 and minus 3.00 per cent across the same ten. The line fitted to them reads 0.5000 plus 1.5000 times the Nakshatra change, and those two coefficientsThe numbers a fitting procedure picks out and drops into a rule whose shape somebody already chose. Here there are two: the number the input gets multiplied by, and the number added on at the end. are exact by construction rather than rounded off. One settled line raises a question nobody asked of it before: of the 218 it misses by, how much could any rule at all have avoided, and how much was never available to anybody?

What is a model, in plain words?

A model is a rule that takes one or more inputs and returns a statement about an outcome. The definition needs to say nothing more. There is no requirement that the rule be complicated, no requirement that it involve arithmetic that cannot be done in the head, and no requirement that it be right.

A tea cart outside a bus depot shows the shape of it. The person running it wants to know how much milk to buy tomorrow, so they use a rule: buy tomorrow whatever sold today. The rule is a model. The rule reads one input, today's takings, and returns one statement, tomorrow's likely takings. The rule ignores the weather forecast, the day of the week, the exam season that empties the depot in March, the roadworks that started on Tuesday and the fact that the cart across the road has been shut for a fortnight. The ignoring is not a shortcut taken by somebody who could not be bothered. A rule that took in every one of those considerations would take longer to run than the day it is trying to predict, so the ignoring is what makes the rule usable at all.

Notice who does what here. A person decided that tomorrow's takings should be read off today's, and a person decided that nothing else would be read. The data was never consulted about either choice. The data can do one thing once that shape is fixed: choose the numbers that sit inside it. If the cart owner had written the rule as "tomorrow equals some number times today, plus some other number", a record of past days could pick those two numbers out. The record could not have proposed the form of the rule, and it could not have volunteered that the weather might matter. Fitting chooses the numbers inside a shape somebody else already chose. The choice of shape is therefore the modelling decision, and the fitting is the arithmetic that follows it.

What is this model deliberately leaving out?

Look again at the line on these ten months. The line says the Vasant unit's change is 0.5000 plus 1.5000 times the Nakshatra unit's change, and inside that short sentence there are two entirely separate decisions doing two entirely separate jobs.

The first decision is about what gets read. The rule reads one column and one column only, the Nakshatra unit's change in the same month. The rule does not read the month before. The rule does not read the second unit sitting in the same record that tracks the first one almost perfectly. The rule does not read the marker that is on in five of the ten months. Every one of those was available and every one of them was left out.

The second decision is about the form the rule takes once those inputs are fixed. Having decided to read the Nakshatra change, somebody then decided that the answer would be that change multiplied by one number with another number added on. Not squared. Not bent. Not stepped. Straight. The choice of inputs and the choice of shape are two decisions and not one, they cost different amounts, and a reader who cannot tell them apart cannot know whether to argue about the shape of the rule or about the inputs it is reading.

Back at the tea cart, the same split is easy to feel. Deciding to look at yesterday's takings and nothing else is one decision. Deciding that tomorrow equals yesterday exactly, rather than yesterday plus a bit on Fridays, is a second one. If the cart is wrong on Fridays because Friday is payday at the depot, no amount of cleverness applied to yesterday's number will fix it. Friday is not in the number. The ten months get the same treatment as the tea cart: the two decisions are priced separately, in the same units, so they can be compared.

What does the straight line shape cost on this record?

One fact about these ten months makes the whole split possible, and it takes about ten seconds to see once somebody points at it. Four of the ten months, months one, five, eight and ten, all show a Nakshatra reading of exactly 1.00 per cent. Their Vasant outcomes are 3.00, minus 3.00, minus 1.00 and minus 3.00 per cent. Top to bottom, that is a spread of six percentage points.

Any rule reading only the Nakshatra unit is sharply limited on those four months. The rule sees 1.00 per cent, 1.00 per cent, 1.00 per cent and 1.00 per cent. Four identical inputs. Whatever rule is written, however it is bent, whatever exotic curve is fitted, it must return the same answer for all four. The rule has nothing to tell them apart with. The best a rule can do on four months it cannot distinguish is to give them one number, and the number that leaves the least squared miss is their average. A vague worry about model quality has turned into an arithmetic question with an exact answer.

Their average is minus 1.00 per cent. Take the gaps to it: 4.00, minus 2.00, nothing and minus 2.00. Square them and add: sixteen plus four plus nothing plus four is 24. So those four months alone put 24 beyond the reach of every rule that reads the Nakshatra unit and nothing else, no matter how clever. The same argument runs twice more. Two months show a reading of minus 4.00 per cent with outcomes of 2.50 and minus 7.50 per cent, whose average is minus 2.50 per cent and whose gaps of 5.00 and minus 5.00 square and add to 50. Two months show 6.00 per cent with outcomes of 18.50 and 6.50 per cent, averaging 12.50 per cent, with gaps of 6.00 and minus 6.00 squaring and adding to 72. The remaining two months are alone at their readings, so they contribute nothing.

Add the three: 24 plus 50 plus 72 is 146. The 146 is not an estimate, an assumption or a rule of thumb. The number is the exact floor on the sum of squared misses for every rule that reads this one input on this record, and it was obtained by groupingSorting a record into piles so that every reading inside a pile is identical on the column used for sorting, then looking at what still varies inside each pile once the sorting has taken away everything it can. the record on the input and adding up what still varies inside the groups. All three figures are in squared percentage points, the unit a sum of squared misses lives in.

So how does the 218 split into two parts?

The straight line misses by 218. No rule reading the Nakshatra unit can miss by less than 146. Subtract, and 72 is what remains: the part of the miss that is there because the shape was chosen to be straight rather than because the input was chosen to be that one column. A richer curve can attack the 72 and can never touch the 146, and knowing which is which is the entire value of doing this arithmetic.

Working it group by group makes the two totals reconcile in both directions, and the check is worth doing before trusting either. Inside each group of tied months, the straight line has to give one answer too, so it is competing on exactly the same terms as any other rule, and the only question is whether it happens to pick the best available answer. On the four months at a reading of 1.00 per cent the line returns 2.00 per cent while the best available is minus 1.00 per cent, so it leaves 60 where 24 was possible and wastes 36. On the two months at 6.00 per cent it returns 9.50 per cent against a best of 12.50 per cent, leaving 90 where 72 was possible and wasting 18. On the two at minus 4.00 per cent it returns minus 5.50 per cent against a best of minus 2.50 per cent, leaving 68 where 50 was possible and wasting 18 again. The two lone months it hits exactly.

Read the bottom row twice. The middle column adds to 146 and the next adds to 218, and the difference column adds independently to 72, so the split was not asserted and then divided up afterwards. The split was built from the record five rows at a time and it landed on the same two numbers. As a share, 146 of 218 is 66.97 per cent and 72 of 218 is 33.03 per cent, so two thirds of what looks like model failure is not failure of the model at all.

What ceiling does that floor put on R squared?

A floor on the miss is also a ceiling on the fit. The two are the same statement written in different units. The Vasant unit's own up and down movement over the ten months, measured as a sum of squared gaps to its own average of 2.00 per cent, comes to 893. R squared is one minus the share of that 893 which the rule fails to explain. The straight line leaves 218, so it reaches one minus 218 over 893, or 0.7559.

Now put the floor in the same expression. The best conceivable rule reading the Nakshatra unit leaves 146, so it reaches one minus 146 over 893, or 0.8365. The 0.8365 is a hard stop rather than an aspiration. No rule of any shape reading this one input can get above it, this year or ever. The four tied months will still be tied whatever anybody fits.

Read the three numbers out of a hundred and the argument becomes uncomfortably concrete. The line explains 75.59. The ceiling is 83.65. The gap between them is 8.06, and that is the entire prize on offer to anybody who abandons the straight shape for something better. The remaining 16.35 sits above the ceiling, so it is not a prize at all, and reaching it needs a different input rather than a different shape. A reader who knows those three numbers can stop arguing about the shape in an afternoon and start arguing about the inputs. The argument about inputs is the one actually worth having.

What is a log return, and how far does this arithmetic need one?

Every figure above was worked on changes written the ordinary way: a move from 100 to 103 is a change of 3.00 per cent. There is a second way to write the same move, and it matters here for one narrow reason. A log return, also called a log change, is the logarithmThe power a fixed base number would have to be raised to in order to reach a given value. The natural one, used here, has a base a little over two and seven tenths and is the one that turns multiplying into adding. of one plus the ordinary change, written back as a percentage. The same move from 100 to 103 becomes 2.9559 per cent instead of 3.00 per cent.

Log changes are worth writing because they add across months where ordinary changes have to be multiplied. Chaining the Vasant unit's ten ordinary changes together, month on month, ends at 1.16856953 times the starting amount, a total of 16.86 per cent over the ten months. The chaining took nine multiplications. Written as log changes, the same ten months simply add up, to 15.58 per cent, and 15.58 per cent is the log change corresponding to that same 16.86 per cent total. Adding is easier than multiplying, and that convenience is the entire reason anybody writes a change this way; how simple, compound and log changes differ from one another in full is covered separately, at the end of these notes.

Now the part that belongs here. Refitting the same line on the same ten months, written as log changes on both sides, moves the coefficient on the input from 1.5000 to 1.4710. The number added on at the end moves from 0.5000 to 0.3112. Nothing about the ten months has changed. Nobody collected new readings, nobody dropped a month, nobody chose a different shape. A decision that looks like formatting moved the headline figure by 0.0290. A model is a rule applied to a column, and rewriting the column rewrites the model.

One caveat is the trap, and it has to be said plainly. On the log version R squared comes out at 0.7629 against 0.7559 on the ordinary version. Anybody would be tempted to conclude that the log version fits better. R squared is a share of the movement in whatever column is being explained, and the column being explained is not the same column in the two cases. The log version does not fit better, and the comparison has no meaning at all. The ordinary version explains a share of 893. The log version explains a share of 839.1377. Two shares of two different totals cannot be ranked against each other, and saying one is higher is the error rather than the finding.

How to evaluate a factor model: which three questions matter?

The phrase factor model sounds grander than it is. A factor model is a model built from a few named inputs, each of which is one measured column, combined into a statement about an outcome. A factor model is a shape and nothing else. The line on these ten months is one, built from a single named input; adding the second unit or the on and off marker would make it one built from two or three. Nothing about that shape says what the columns should be about.

Three questions evaluate such a model, and this one record answers all three without needing anything new collected.

Question one, how much does it explain? R squared 0.7559 on the ordinary changes. Everybody asks that question first, and it is the easiest of the three. A single number that goes up when things improve is comfortable to look at. Answering it well proves very little on its own.

Question two, is what is left over patternless? Here the record says no. The ten misses, in time order, are 1, 9, 8, nothing, minus 5, nothing, minus 3, minus 3, minus 2 and minus 5, and their lag one autocorrelationHow strongly each month's leftover gap resembles the gap in the month immediately before it, on a scale where nought means no resemblance at all and one means a perfect echo. is 0.4862. In words: a miss above the line tends to be followed by another miss above the line. Leftovers with a shape to them mean the rule is still failing to capture something systematic rather than merely bumping into noise, and no amount of R squared makes that go away.

Question three, do the inputs say the same thing as each other? On this record two of the available columns do. A second unit sits alongside the Nakshatra unit tracking it almost perfectly, and its variance inflation factorA count of how much wider the uncertainty around one learned number becomes because another input in the same rule is carrying nearly the same information. One means no widening at all. is 15001. Fitting both together, the two numbers the rule learns are minus 1.0000 and 2.5000. The two numbers still add to 1.5000, so the pair together says exactly what the single input said while each half of it says something absurd. Duplication between two inputs is covered separately.

A model that passes only the first question has passed the easiest one, and this model passes the first and fails the other two. None of that is a verdict on whether anybody should do anything; it is a description of a rule and its leftovers.

What should be asked of any model before its output is read?

Four questions, in this order, and none of them requires refitting anything or understanding how the rule was built. The four questions work on a spreadsheet handed round in a meeting as well as they work here.

First, what is it reading? Get the list of columns. If nobody can produce the list in under a minute, that is itself the finding. Second, what is it ignoring? Take the columns that exist in the same record and are not on the list, and ask why each one is off. On these ten months the answer is that the second unit, the on and off marker and the previous month were all available and all left out.

Third, which part of its error could a better shape reach and which part could not? The floor calculation answers that, and it needs only the input column, the outcome column and the ability to sort. Fourth, what is the ceiling on how well anything reading these inputs could do? The ceiling is the same floor expressed as a fit rather than as a miss.

Here that last pair is the whole story. The ceiling on this record is 0.8365 and the model sits at 0.7559, so the argument about shape is worth 8.06 read out of a hundred and the argument about inputs is worth everything else. Somebody reading a lender's scoring rule, an analyst reading a demand forecast, a household comparing two electricity tariffs on last year's bills: all three are looking at a rule that reads some columns and ignores others, and in all three cases the same four questions separate a rule that is badly built from a rule that is well built on inputs that cannot carry the weight. A badly built rule and a well built rule on inputs that cannot carry the weight look identical from the outside and have completely different fixes, and only the floor calculation tells them apart.