Structured Data vs Unstructured Data: What Each Answers
Two records sit in the same small office. One is a ruled file the clerk types up every month, and it will report in four seconds what the market took, how many stalls filed and what the average came to. The other is a paper notebook the same clerk writes in by hand. There is nothing in it to add, so it reports none of those things. Almost everybody handed both of these treats the first one as the real record and the second one as a nice touch. On this record that instinct is wrong by Rs 14,896.55/-, and the gap opens at a single cell.
The Neelbagh stall record. A covered market, a market office, and that office keeping one row per stall per month across four months. The file it hands over is 32 rows and 8 columnsA vertical strip of a ruled record. Everything down it is supposed to be the same kind of thing, and the heading at the top says which kind., so 256 cells, and 31 of the takingsThe money a stall took over its counter in a stated period, counted before anything at all is paid out of it. cells carry a number. Rows, columns and written column lists are covered separately and are taken as given here.
The two cells that read Rs 4,80,000/-. NB-07 Sundari Chaat and NB-08 Peeli Mithai both have Rs 4,80,000/- in their month 3 takings cellOne box in a ruled record, where a single row meets a single column. It holds one value and nothing else.. One of those is a real month and one is a typing error. Establishing which is which is covered separately, and the pair is used here as it stands.
The counting rule, named every time a figure appears. The average takings per stall month is the sum of the takings cells that carry a usable number, divided by how many carry one. Three rulings ride along with it and all three belong to the office rather than to arithmetic. A blank holds nothing at all, so a blank is not a number. The office code for no return received is a label wearing digits rather than an amount, and a label is not a number either. And a real zero is a number. Somebody wrote the zero down on purpose. The three rulings are a convention rather than a discovery, so they travel beside every figure below.
What actually makes a record structured?
A record is structured when the meaning of every value is fixed before the value is written, by the column it is going to sit in. There is nothing more to it than that. Somebody decided in advance that this strip holds a stall identifier, that strip holds a month, and this one holds a rupee amount of takings. Once those decisions exist, any single cell can be understood without being read in context. Its column already said what the cell would mean. The test for structure is whether a cell's meaning can be stated without reading it.
The definition mentions nothing about tidiness, nothing about file formats, and nothing about how the record is stored. A notebook kept in beautiful handwriting, ruled with a ruler, dated on every sheet and never blotted is still unstructured. Knowing what any figure in it means requires reading the sentence around it. A table somebody typed up quickly with a coffee ring on the printout is structured. The column heading above each cell did the deciding long before the typing started. Neatness is a property of the handwriting. Structure is a property of the decisions.
Here is the everyday version, and it is one most households already run. Think of a kitchen shelf with three labelled jars: rice, dal, sugar. The label settled the question before anything went in, so anybody can walk in, reach into the middle jar and know what they are holding without tasting it. Now think of the same kitchen with one large unlabelled tin into which everything gets tipped, along with a note taped to the lid explaining what went in on which day. The tin may hold more, and the note may be more honest and more detailed than three labels ever could be. But nobody can answer how much sugar is in the house without reading the note through from the beginning.
Two records land on the desk. One is a beautifully written notebook in a careful hand. The other is a scruffy table with a coffee ring on it. Which one is structured, and what settles it?
What can the Neelbagh stall record answer by itself?
With the record on the desk, the asking can start. How many stalls do its rows name? Counting the different entries in the stall identifier column gives eight. How many months does it cover? Counting the different entries in the month column gives four. What do the month 1 rows add up to? Adding those takings cells gives Rs 3,08,000/-. What is the average takings per stall month as the file was handed over, being the sum of the takings cells that carry a usable number divided by how many carry one? Adding the 31 cells that carry a number and dividing by 31, the average comes to Rs 69,035.45/-.
Every one of those four answers arrived by counting or by adding, and not one of them required a human being to read the record and form a view. Answering without reading is not a small property, and it is the reason the ruled file exists at all. The interpreting happened when the columns were named, before the first row was typed. A clerk who has typed the rows correctly has already done all the interpreting that will ever be needed.
The record is asked how many stalls its rows name. What does it actually have to do to answer that?
Two honest cautions, both of which belong elsewhere. Whether seven month 3 rows means seven stalls really traded is a question about what is missing from the file, and that is covered separately. Whether counting the stall name column instead of the identifier column gives the same answer is a question about duplicates, and that is covered separately too. Neither of them changes what a fixed column permits without reading.
What is the day book, and why is it not structured?
The same market office keeps a second record, and hardly anybody ever asks for it. The day book is a paper notebook in which the clerk writes down what happened, in sentences, in the order things happen. Seven returns at the desk by close of day. Rain from noon and the fruit stalls slow all afternoon. Sundari Chaat took the wedding order for the hall on the main road, cooked over two days, paid in one go on the day it was collected. There are no columns. There is no rule anywhere saying what any part of any sentence is.
Ask that notebook what the market took in month 1 and it yields nothing. Not a wrong answer, not an approximate answer. The notebook holds no set of things to add, so it gives nothing at all. Numbers do appear in the sentences. Nothing marks which of them is a rupee amount and which is a day, a pitch measurement or a day book folio. The day book cannot be added up, and that is no defect in the day book: it was never built to be added up. Every record answers the questions its shape was built for and goes silent on the rest, and holding one to the other one’s standard is how an office ends up binning the record that could have saved it.
The market office day book is taken to the desk and asked what the market took in month 1. What does it give?
Which of the two answers each question?
Set four questions against both records and the pattern shows up immediately. How many stalls do the month 3 rows name? The record counts them and answers seven; the day book cannot count anything. What do the month 1 rows add up to? The record adds them and answers Rs 3,08,000/-; the day book cannot add anything. Which return reached the office latest? The record answers day 19 from its filed on day column, and the day book, being a diary of the desk, may well have that too. And then the fourth. Why does NB-07 Sundari Chaat’s month 3 cell read Rs 4,80,000/-?
The record cannot answer the fourth question and the day book can, and the fourth question is the one that decides whether the first three answers are worth anything. The asymmetry is the whole comparison in one line. Three questions go one way, one goes the other, and the traffic is nothing like even in importance. The three the record answers are arithmetic about what is in the file. The one it cannot answer is about the world the file came from, and no column was ever set aside to hold the world.
Four questions, two records. Which of these three goes to the day book and not to the record?
Sort eight questions into the record that can answer each one
The difference between a ruled table and a written notebook is a difference in kind rather than a quantity that varies, so what the panel below offers is a sorting job rather than a setting to slide. Eight questions, three trays, and every tray starts empty. Each question lands in a tray with an answer beside it, and the shape of the comparison builds itself. One of the eight straddles, and that is exactly why both records get checked rather than deciding by category which one to walk over to.
Educational illustration. The Neelbagh market, its stall record, its market office and its day book are inventions built for teaching, and every figure above is illustrative. Every figure printed uses the counting rule named in the opening panel, and no other denominator appears anywhere. A person decided every routing above and wrote it down, and that is the honest description of how a question actually gets sent to the right record.
Which return reached the market office latest, and which of the two records can answer that?
What is one sentence in a notebook worth?
Two cells in this record read Rs 4,80,000/-. The two cells sit in the same column of the same month, they hold the same six digits, and nothing inside 256 structured cells separates them. One of them is NB-07 Sundari Chaat, whose other three months are Rs 40,000/-, Rs 41,000/- and Rs 42,000/-. The other is NB-08 Peeli Mithai, whose other three months are Rs 42,000/-, Rs 45,000/- and Rs 47,000/-. Look only at the takings column and the two cells are twins.
The fact that separates them is not in the record at all. The day book carries one sentence recording that Sundari Chaat took a wedding orderOne large catering order placed for a wedding and settled in a single payment, worth many ordinary days of counter trade. for the hall on the main road and was paid in one go on the day it was collected. Peeli Mithai’s own return slipThe small paper a stall fills in itself and sends to the office, declaring its own figure for the month. for that month reads Rs 48,000/-, and the office’s own working on the month’s stall fees agrees with Rs 48,000/-. The day book sentence and the return slip are why one cell is corrected and the other is left standing. The two cells differ in their provenanceThe paper trail behind a figure: which document it was copied off, and whose hand wrote that document. and in nothing else whatsoever.
Now put a price on it. Correct NB-08 Peeli Mithai’s month 3 cell to Rs 48,000/- and leave NB-07 Sundari Chaat’s alone, and the average takings per stall month, being the sum of the takings cells that carry a usable number divided by how many carry one, goes from Rs 69,093.10/- down to Rs 54,196.55/-. The correction moves a published figure by Rs 14,896.55/-. One sentence in a notebook that cannot be added up is worth Rs 14,896.55/- on a figure that can. Double counting is easy at this point, so say plainly what the quantity is: the worth of the sentence and the move the correction makes are one quantity under two names, not two separate findings. The sentence is the only reason the correction happens, so they are the same number seen from two sides.
One sentence in the day book records a wedding order at NB-07 Sundari Chaat. What is that sentence worth here?
What does turning free text into a column throw away?
The obvious repair, once this is seen, is to structure the notebook. Suppose the office adds a ninth column called reason for unusual month, and settles in advance the entries it is allowed to take: wedding order, festival week, pitch shut. Now the sentence about Sundari Chaat fits into a cell, and for the first time somebody can count how many unusual months in the year were wedding orders. The gain is real, and it should be said clearly before the cost is.
Then next year happens. The road outside is dug up for nine days. A stall changes hands halfway through a month and files two part returns under one keyThe small set of values meant to pick out one row and no other, such as a stall and a month taken together.. Rain from noon leaves the fruit stalls slow all afternoon. The allowed entries were written by somebody in a room in a particular year, and no such person has ever produced a complete list, so none of those three has a home among them. A column can only hold what somebody anticipated, so structure buys counting and pays for it in everything nobody thought of.
The everyday version is a household budget with a line for rent, a line for groceries and a line for travel. The three lines work for years. The budget has nowhere at all to put the month the fridge died, and the person keeping it does one of two things: invents a line, or squeezes the fridge into groceries. The second is worse than having no line at all. Squeezing the fridge into groceries produces a grocery total that is countable, tidy and false. The office faces the same choice with its allowed entries, and the honest answer is to add the column and keep the notebook, rather than to add the column and pretend it replaced the notebook.
The market office adds a column called reason_for_unusual_month, with a fixed list of allowed entries settled in advance. What has it gained and what has it lost?
Is countable the same as complete?
The record is 256 cells and every one of them is countable. The day book has no cells at all and holds the fact that decides the hardest question in the file. Ask which of the two is the more complete record of what the office knows, and the honest answer is neither. The two are complete about different things. The record is complete about totals and silent about reasons. The notebook is the reverse.
Countable is not the same as complete, and the reader who forgets it will be right about every total and wrong about the one thing that matters. Nobody makes the mistake deliberately, so the difference is worth keeping as a habit rather than as an observation. Nobody decides the notebook is worthless. Somebody is handed a file, the file is obviously the record, the file answers everything asked of it, and what else the office writes down simply never comes up. The notebook is not rejected. Nobody mentions it at all.
The record has 256 countable cells and the day book has none at all. Which of the two is the more complete record of what the market office knows?
What does an office do when the deciding fact is in the notebook?
A notebook cannot be added up and must not be thrown away, so the practical move joins the two rather than converting one into the other. One narrow column is added to the record, holding a day book folio, the number written at the head of one leaf of the notebook. The column is filled in only for the rows somebody has written a note about, and most rows leave it empty. NB-07 Sundari Chaat’s month 3 row gets folio 41. The extra column is the entire change.
A pointer costs one narrow column, requires no text to be read by any machine, and would have made both cells reading Rs 4,80,000/- traceable in seconds. The record stays countable. A folio is just another value in a fixed column. The notebook stays whole. Nothing in it has to be shortened, categorised or rewritten to fit. And the two stop being separate objects in separate cupboards: the moment one of the twin cells carries a folio and the other does not, a person looking at the takings column has a reason to walk to the shelf.
Every kind of reader benefits from the same move, and that is what makes the column worth adding. Somebody lending against a stall’s takings wants to know whether an unusually strong month repeats or was a one off, and the folio takes them straight to the sentence that says. Somebody buying a stall as a going concern wants the same thing before pricing it. Somebody in the office defending last year’s published average wants to show where each correction came from, and a folio is the shortest possible way of showing it. In every one of those cases the reader is asking not for the number but for where the number came from, and a structured record answers that only if somebody built it a doorway.
The office adds one narrow column holding a day book folio for any row somebody has written a note about. What does that cost, and what does it buy?
The failure: a careful analyst, every total correct, and the answer wrong either way
Somebody is handed the Neelbagh stall record and told it is what the market office knows. The analyst is careful. The analyst guesses at nothing. Every total they compute is correct. Inside 256 structured cells there is no way to treat two identical figures differently, so the analyst reaches month 3, finds two cells reading Rs 4,80,000/-, and treats both the same way. Asking what else the office writes down never occurs to them. The day book is on a shelf about three feet behind them.
Watch what that costs. The cost is not a caution, it is a pair of numbers. Treat both cells as real months and the average takings per stall month comes out at Rs 69,093.10/-, or Rs 14,896.55/- above the honest figure. Treat both as typing errors and knock NB-07 Sundari Chaat’s month back to a typical Rs 40,000/-, and it comes out at Rs 39,024.14/-, or Rs 15,172.41/- below it. The honest figure, Rs 54,196.55/-, sits between them and is not reachable from inside the record at all. Whichever way a careful person handles the pair, one of the two cells is handled wrongly, and the published average is out by a stated amount in one direction or the other.
One question closes the gap, and it is asked before the file is asked for: what else does the keeper write down? Not because the answer is often useful, but because the occasions on which it is useful are exactly the occasions nothing inside the file can flag.
Covered elsewhere. Rows, columns, types and written column lists are covered earlier and are taken as given here. Deciding whether an unusual figure is an error rather than a real observation is covered later, and the settled outcome for the two cells reading Rs 4,80,000/- is used here as it stands. Getting a tool to read text, and every technique that grows out of doing so, belongs with the programming notes: not one word above was searched, counted or machine read, and the whole comparison stands without any of it.
What the comparison settles. A notebook is not better than a table, and an office keeping one is doing nothing wrong. The two records answer different questions: three of the four questions above go to the table, and the fourth decides whether the other three matter.
Where did these figures come from?
Four things produced every figure above: the stall record, the day book, the counting rule, and the two documents that settle the twin cells. The table below names each one and where it can be checked.
| What was used | What it is | Where it can be checked here |
|---|---|---|
| The Neelbagh stall record | An invented office file of 32 rows across 8 columns, one row per stall per month over four months | Its month 3 rows are printed in the pointer block, and its column names in the first illustration |
| The market office day book | An invented paper notebook of sentences kept alongside the file by the same clerk | Three of its lines are printed in the first illustration and one of them is quoted again in full |
| The counting rule | Which takings cells count towards the average and which stay out of both halves of it | Stated in the opening panel and repeated beside every figure it produces |
| The two documents outside the record | One day book sentence about a wedding order, and one stall return slip reading Rs 48,000/- | Both drawn in the illustration of the twin cells, with what each one settles |
The Neelbagh market and its fourteen pitches, the Neelbagh stall record, the market office, the clerk’s day book, and the stalls NB-01 Kadamba Idli, NB-02 Chandan Tea, NB-03 Harit Greens, NB-04 Peetal Utensils, NB-05 Bansi Flour, NB-06 Ilaka Fruit, NB-07 Sundari Chaat, NB-08 Peeli Mithai, NB-09 Roshni Juice and NB-10 Amber Rolls are invented.
Educational material. Not advice on any investment, tax, budget or market position.