The Dataset: Rows, Columns, Types, Fields and a Written List
A dataset is a grid: one row for each thing observed, one column for each fact recorded about it, and one value in each cell. The Neelbagh stall record, invented, has 32 rows and eight columns, so 256 cells. A dataset becomes usable not through the grid but through the written list beside it, saying what each column holds, in what unit, and what an empty cell means.
Two operations do all the work of reading a record, and nothing else is needed. Counting, and division. Reading a record never judges whether a value is large or small or surprising, so no rule about spread, no curve, no fitted line and nothing estimated comes into it. Reading a record says what a value is, and what the column it sits in has promised about it. Saying what a value is turns out to be a narrower job than it sounds, and it is the job that goes wrong first.
Every count and every rupee figure below was recomputed from the 32 rows the market office handed over. The Neelbagh market, the Neelbagh stall record and all ten stalls in it were built for teaching and exist only here. All 32 rows are set out below, so the addition can be checked by hand.
What is a dataset, and what does one row of this record hold?
Picture the market office. Ten stalls trade in the Neelbagh covered market, each one files a small monthly returnThe short form a trader hands in each month telling the office what the month brought. It is a declaration made by the trader, so the office is copying what it was told rather than measuring anything itself. saying what it took, and a clerk types those slips into one file. Rows go down the file and columns go across it. Every row is one thing observed. Every column is one fact recorded about every row, always in the same place, always written the same way.
The hard part is never the grid; it is agreeing what one row is. In the Neelbagh stall record a row is not a stall. A row is a stall in a month. NB-01 Kadamba Idli appears four times, once for month 1, once for month 2, once for month 3 and once for month 4, and those four rows are four different things observed. Settle what one row stands for before anything else in a record is touched. Everybody who reads the file later will assume whatever the file looks like it means, and half of them will assume wrong.
The record the office hands over has 32 rows and eight columns. Eight times 32 is 256, so there are 256 cells in it, and 31 of the 32 takings cells carry a number. The eight columns, spelled exactly as the office spells them, are stall_id, stall_name, licence_no, category, month, takings_rupees, pitchThe floor space a market gives one trader. In a covered market the office measures it out, numbers it and rents it, and a stall that grows has to be moved rather than expanded._sqft and filed_on_day. Every later check is written against those exact spellings, so the spellings matter more than they look.
In the Neelbagh stall record, what is one row?
What is a Data Point, and what is the smallest thing in a record that means anything?
One cell is one observationOne recorded reading of one thing at one time. The word carries no promise that the reading is right, only that somebody wrote it down.. A single cell is also the smallest thing in the whole file that means anything on its own, and it has a name: a data point. Take NB-07 Sundari Chaat, month 3, the takings cell. The cell reads Rs 4,80,000/-. One value in one named cell is one data point.
A data point is a value together with its position, and neither half is optional. Somewhere else in the same record, NB-08 Peeli Mithai's month 3 takings cell also reads Rs 4,80,000/-. Identical value. Different data point entirely. The cell sits in a different row, so it says something about a different stall. Copying the number out of the file into a note that says only Rs 4,80,000/- does not copy a data point. The note copies half of a data point and throws the other half away, and there is no way to get it back.
Think of an electricity meter. A reading of 4,182 units means nothing at all until somebody says which meter it came off and on which day. The number is the easy half. The position is the half that gets lost, and it gets lost every time somebody pastes a figure into a message.
Two cells in the record both read Rs 4,80,000/-. Are they the same data point?
What is a Variable, and what does a column vary across?
A column is a variable: one fact, recorded the same way for every row in the file. The definition is that short, and on its own it is not very useful. The useful question is the second one. Ask what each column varies across, and eight columns split cleanly into three groups.
Five of them should take one value for a stall whatever the month. NB-01 is NB-01 in all four months. Its name, its licence number, its category and its pitch area do not change from month 1 to month 4. None of the four is a monthly fact about the stall. One column, month, takes one value for a month whatever the stall: every row of month 3 reads 3, and it says nothing about which stall the row belongs to. And two columns vary across both, takings_rupees and filed_on_day. Each of those is a fact about one stall in one particular month and cannot be written down without naming both.
Splitting the columns this way is not bookkeeping. The split is a check, and the cheapest check there is. The file is grouped by stall_id, and the number of different values each stall-level column takes inside each group is counted. Every count should come back as one. With stall_id itself set aside as the label being grouped by, four columns are left to test. Three of those four come back clean, and stall_name does not: NB-03 has two names in the file, written Harit Greens in months 1 and 2 and Harit Green in months 3 and 4. Notice what found it. Not a spelling rule, not a dictionary of stall names, not a person reading carefully. A count inside a group, done by a tool in a second and needing no knowledge of the market at all.
Which stall-level column in this record breaks the promise of one value per stall, and how would it be found?
What is a Data Type, and what exactly does a column promise?
A data type is what a column promises about the kind of thing sitting in it: text, a whole number, an amount of money, a date. The promise is short and it is mechanical. A column declared to hold whole numbers will be sorted, added, subtracted one value from another and averaged by a tool, with no question asked of anybody. The same tool will sort a column declared to hold text alphabetically, and then refuse to add it. Sorting and refusing is the entire service a type provides, and the service is genuinely useful. The machinery works the same way on every file anyone ever hands over.
Now the trap, in one line. A tool checks the type and never the meaning, so a column can keep every promise its type makes and still be complete nonsense. Nothing in the type of a column knows what the numbers in it are counting, or whether they are counting anything at all. The unitThe thing a number is counted in: rupees, square feet, days, kilograms. Two numbers can only be compared or added when both were counted in the same one. is not part of the promise. Whether the column is a measurement or a label is not part of the promise either. Both of those live somewhere else, and if nobody wrote them down, they live nowhere.
What happens when an identifier is stored as a number?
Here is the record's own instance of that trap, and it is worth walking slowly because it is almost invisible. The column licence_no holds a four digit number the market office issues to a stall when it first takes a pitch. A licence number is an identifierA code whose only job is to point at one thing and tell it apart from every other. It is written down so it can be matched and looked up, never so it can be added to anything., not a quantity. Nobody has 2,104 of anything.
The column sorts, and the sort is not even meaningless: a lower licence was issued earlier, so sorting the column puts the stalls in the order they joined the market. Joining order is a real fact to lean on. Then it averages. The ten licence numbers in the market are 2104, 2216, 2318, 2405, 2477, 2530, 2618, 2743, 2801 and 2896. The ten add to 25,108. Divided by ten, that comes to 2510.8. The average is arithmetically perfect and describes absolutely nothing. The lowest subtracted from the highest gives 792, and 792 is not 792 of anything either.
The damage gets worse. Somebody builds a summary that averages every numeric column by category, and it prints 2590.25 for cooked food and 2508.50 for beverages. Neither figure is money, so neither carries a rupee sign, and both sit in a table beside figures that do. A reader looks at that table and takes all of it for facts about the market. Nothing complains, ever. No tool refuses, no warning appears, no cell turns red. The only thing on earth that catches this is a written list saying licence_no is a label.
House numbers do the same thing. The numbers taken off the doors along any street will average out to a number. Nobody lives there.
The average licence number is 2510.8. What is wrong with that figure?
What can be right in type and still wrong in unit?
The column pitch_sqft holds whole numbers, and every single value in it is a whole number. The type is kept, completely and without argument. Now look at NB-05 Bansi Flour. Its pitch is recorded as 14. The column is measured in square feetA measure of floor area: the area of a square whose sides are one foot long. Ten square metres is a little over a hundred and seven square feet, so the two measures are nowhere near interchangeable., and NB-05's pitch is really 150 square feet. Somebody, somewhere, wrote down square metres.
A 14 square foot stall is a stall nobody could stand up in with a sack of flour. Nobody caught it. Nothing about the number 14 breaks a promise: the value is whole, it is positive, it sorts, and it sits quietly in a column of whole numbers. The damage arrives the moment somebody divides by it.
Take month 1, where every takings figure in the file is clean, and work out takings for each square foot for the eight stalls. NB-05 took Rs 55,000/- in month 1. Divided by its recorded 14 square feet, that reads Rs 3,928.57/- for each square foot, and NB-05 comes out top of the table by a distance no other stall gets close to. Correct the pitch to its true 150 square feet and the same takings read Rs 366.67/-, fifth of eight. The stall that really tops that table is NB-02 Chandan Tea at Rs 516.67/-, and it does so because it has the smallest pitch in the market at 60 square feet.
Four places, from one cell. And here is the part worth sitting with: the table sorted correctly and averaged correctly both times. Nothing failed. There was no error message to miss, no cell to notice, no warning to ignore. A right type with a wrong unit produces a table that is perfectly well behaved and perfectly wrong, and the only place the fault could ever have been caught is a written line saying what pitch_sqft is measured in and roughly what size a pitch is.
NB-05 Bansi Flour is recorded with a 14 square foot pitch and tops the table of takings for each square foot. Correct the pitch to its true 150 square feet. Where does NB-05 land?
Move one recorded pitch and watch eight bars change places.
One control, and it moves one number: the pitch area recorded against NB-05 Bansi Flour, anywhere from 10 to 200 square feet. Month 1 takings are held fixed for all eight stalls, and the other seven pitches are held fixed too. Three things redraw. The tallest bar always fills the panel, so the bars rescale, and the scale is written on the panel. The bars are drawn in rank order rather than in stall order, so they physically change places. And the dashed line marking NB-02 Chandan Tea at Rs 516.67/- stays put, so NB-05 can be seen crossing it. The control starts at 14 square feet, the pitch as the office recorded it, reproducing the Rs 3,928.57/- and the first place printed above.
pitch_sqft holds only whole numbers, and NB-05's recorded value of 14 is a whole number. Has the column kept its type promise?
What is a Data Dictionary, and what must it say about every column?
Everything above has the same shape. A tool did exactly what it was asked, and the answer was still worthless, and the only thing that could have prevented it was somebody writing down what the column actually holds. The written list has a name: a data dictionary. A data dictionary sits beside a record and says, for every column, seven things.
Its exact name, spelled as the file spells it. Its meaning, in one sentence. Its type. Its unit of measureThe scale a number was counted on, stated so plainly that a stranger cannot guess wrong: rupees rather than lakh, square feet rather than square metres, whole days rather than dates., if it has one. Its allowed values, if there is a fixed list. Whether it is a label or a quantity. And what an empty cell in it means.
The last two are the two nobody writes, and they are the two that would have caught both faults in the Neelbagh stall record. The file more or less supplies the name and the type, so everyone writes those. Almost nobody writes the label question, and so licence_no gets averaged. Almost nobody writes what emptiness means, and so an empty takings cell is silently read as a zero by one person and as a missing month by the next, and the two of them produce different answers from the same file and never find out why.
Which two lines of a data dictionary does almost nobody write?
How to Document a Financial Dataset: what do those eight lines catch?
The list above is the whole document. Eight lines, one for each column, written before the first summary is built rather than after the first argument about it. Held against what is wrong with this record, the list shows what the habit saves, and saving something is the only thing that makes a habit worth having.
The Neelbagh stall record carries nine deliberate faults. A missing row, an empty cell, a placeholderA stand in value typed into a cell where the real one was not available. It is written in the same ink as everything else, which is precisely what makes it dangerous. that looks like money, one stall filed twice for the same month with different figures, one stall written under two spellings, two cells reading Rs 4,80,000/- of which one is real and one is a typing slip, a pitch in the wrong unit, an identifier stored as a number, and two closed stalls the office deleted without saying so. The eight line list catches three of the nine before anybody opens the file, and it is worth being exact about which three and why.
| What the list says | What that catches | How the check runs |
|---|---|---|
| licence_no | The identifier averaged as though it were a quantity | The line says label, so every summary skips the column. No reading required. |
| pitch_sqft | NB-05's pitch of 14 in a square feet column | The line says expected between 50 and 200, and 14 is outside it. The check is a comparison. |
| takings_rupees | The cell reading 99999 taken for Rs 99,999/- | The line says 99999 is the no return code, so it is never counted as money. |
| takings_rupees | The empty cell in NB-05's month 2 row | The line says what emptiness means, settling the argument without filling the cell. |
| stall_name | The two spellings of NB-03 | Needs the file itself. The line warns against joining on the column, and nothing more. |
| Eight lines | Three faults caught outright | One argument settled, and five faults that need the record read |
Three out of nine is not a rescue. All three were caught by a sentence somebody wrote from knowledge that was never in the file to begin with: what a licence is for, how big a pitch is, and what the office writes when nothing arrives. None of those three is in the numbers, so no amount of staring at the numbers recovers any of them. The rest of the record's faults need the record itself read carefully, and that is covered separately.
What does the whole record look like when the parts are put together?
The Neelbagh stall record end to end, in the order this guide built it, so the numbers stand together rather than scattered. Every one of these figures was recomputed from the 32 rows rather than carried over from anywhere.
| What was counted | The figure | How it was reached |
|---|---|---|
| Rows the office handed over | 32 | Eight stalls over four months, less one absent row, plus one stall filed twice |
| Columns | 8 | stall_id through filed_on_day |
| Cells in the file | 256 | 32 rows times 8 columns |
| Takings cells carrying a number | 31 | 32 rows less the one empty takings cell |
| Distinct stall codes in the file | 8 | Counted from stall_id |
| Distinct stall names in the file | 9 | Counted from stall_name, and the extra one is the second spelling of NB-03 |
| The ten licence numbers added up | 25,108 | 2104 through 2896, added |
| Their average | 2510.8 | 25,108 divided by ten. Not money, so no rupee sign |
| NB-05, month 1, for each recorded square foot | Rs 3,928.57/- | Rs 55,000/- divided by the recorded 14 |
| NB-05, month 1, for each true square foot | Rs 366.67/- | Rs 55,000/- divided by the true 150 |
| NB-02, month 1, for each square foot | Rs 516.67/- | Rs 31,000/- divided by 60, the smallest pitch in the market |
| Average takings for each stall month, as handed over | Rs 69,035.45/- | Rs 21,40,099/- divided by the 31 cells that carry a number |
The last row is the headline the office would print, and it comes with a convention attached that has to travel with it every time. Only those takings cells holding a figure that can actually be used are added up, and that total is divided by the count of the cells added. An empty cell contributes nothing and counts for nothing, so it stays out of both halves. The office placeholder standing in for a slip that never came is not money either, and it stays out of both halves too. A genuine zero is different: it is a real figure, it belongs in the total and in the count, and NB-04 has one. Its pitch was shut through month 2 while it was re-roofed. Changing any part of that recipe changes the figure, so any treatment working to a different divisor states which one in the same breath as the number. Rs 69,035.45/- is itself a rounded display of a figure that does not end.
The file has 8 distinct stall codes and 9 distinct stall names. What does the gap of one establish, without a single name being read?
The summary that averages every numeric column
The market office builds a summary of the record for its own committee. The summary is one table, one row, four figures: average takings, average pitch size, average day filed and average licence number, 2510.8. Every figure is formatted the same way, in the same font, to the same number of decimals. Three of the four are facts about the market. The fourth is arithmetic performed on a label, and the reader has no way at all to tell which one it is.
The same summary then ranks the stalls by takings for each square foot and names NB-05 Bansi Flour the most productive stall in the market at Rs 3,928.57/-. The true figure is Rs 366.67/-, fifth of eight, and the stall that really tops that table is NB-02 Chandan Tea at Rs 516.67/-. The committee now believes two things about its own market that are not so, and neither belief came from a broken tool. Both came from a tool doing exactly what it was told.
Here is the habit that fixes it, and it is one line long. Write the column list before the first summary, and mark every single column in it as label or quantity. A meaningless figure is more dangerous than a wrong one. A wrong figure gets queried; a meaningless one gets quoted.
How a lender actually uses this, before reading a single number
A small business lender is handed a shop's own sales file in support of a loan application. The first thing an experienced credit officer asks for is not the totals. The first request is for the column list, and if there is not one, the officer writes it with the borrower sitting there, one line for each column, out loud.
The reason is entirely practical. Almost every dispute that follows a lending decision turns out to be a disagreement about what a column meant rather than about what a number was. Was that figure billed or collected. Was it inclusive of tax. Does an empty month mean the shop was shut, or that nobody typed it in, or that the file simply stops there. Does that identifier column point at a customer or at an invoice. Ten minutes with a pen settles all of it before any number is trusted, and it costs the officer nothing.
The same ten minutes is what a household should spend on its own spending file, an analyst on a file inherited from somebody who has left, and anybody at all on a file that arrived by message with no note attached. The list is the cheapest insurance in the whole business of working with records, and the only reason it goes unwritten is that writing it feels like a delay.
What stands behind the figures in the Neelbagh stall record?
The faults had to sit exactly where a lesson needs them, so every rupee figure, every pitch area and every licence number in the Neelbagh stall record was built rather than found.
| Source | Document | Site |
|---|---|---|
| The 32 rows themselves | The Neelbagh stall record, built for teaching and printed in full above | None. Printed above |
| Ordinary record keeping practice | None. No single document defines what a written column list must contain | None |
The Neelbagh market, the Neelbagh stall record, the market office, its day book and its ten stalls, NB-01 Kadamba Idli, NB-02 Chandan Tea, NB-03 Harit Greens, NB-04 Peetal Utensils, NB-05 Bansi Flour, NB-06 Ilaka Fruit, NB-07 Sundari Chaat, NB-08 Peeli Mithai, NB-09 Roshni Juice and NB-10 Amber Rolls, are invented.
Educational material. Not advice on any investment, tax, budget or market position.