Python for Finance: What It Is Used For, and the Table You Work In

Python here is one thing: a way to hold a record as a table in memory and ask it a question. The Neelbagh stall returns sit as 32 rows and 8 columns. Grouping those rows by the stall gives 8 groups. Grouping them by the stall name gives 9, and the ninth is a spelling rather than a stall.

Underneath that answer sit three things and nothing more. The first is the Neelbagh stall record itself, an office file covering four months of takings at an invented covered market of 14 pitches, along with the faults it already carries and the counting rule its takings figures answer to. The second is what one row of that file is a statement about. One row is one stall in one month. The third is counting. The counting never runs heavier than that: a count of rows, a count of columns, a count of groups, and one average that always travels with the count sitting underneath it.

What is Python used for here?

Four things. Reading a record in. Checking that all of it is there. Shaping it so it answers the question asked. Answering that question and printing the count the answer rests on. Reading, checking, shaping and answering are not a summary of a longer list. The four are the whole list.

Syntax for its own sake, loops explained because loops exist, keeping a value in a name, installing anything, and how software is packaged and shipped are all real subjects, taught extremely well by people whose job that is. None of them is what an analyst was hired for.

The everyday version is a stall keeper buying a calculator. There is a manual in the box with sixty functions in it, and the keeper learns four keys: add, subtract, the total, and clear. Not because the other fifty six are worthless, but because the day's takings need four and learning the rest before opening the stall is a way of never opening the stall. An analyst who has to answer what the average monthly takings were is in exactly that position on the first morning.

What is a DataFrame, and what does the table hold that the file did not?

A DataFrame is rows and columns held in memory, with a name for every column, a type for every column, and a position for every row. The definition ends there, and it is worth reading twice. The three things a DataFrame promises are exactly the three things a file does not promise. The table the analyst works in is the file plus a declaration about what is in it.

The Neelbagh returns sit in a DataFrame as 32 rows and 8 columns, making 256 cells. The eight column names are not invented by anybody working on the record. The eight names are read straight off the header rowThe first line of a data file, carrying the column names instead of any reading of its own. The header row describes the lines below it and is never one of them. of the file the market office handed over. The names stall_id, stall_name, licence_no, category, month, takings_rupees, pitch_sqft and filed_on_day are the office's words rather than the analyst's.

Two of the three promises change the work, and it pays to be precise about which two. The first is that the types are real, so a column can be told what kind of thing it holds before anybody computes with it. The column licence_no is the clean case. The values 2104 and 2318 and 2743 are digits and nothing else, and every value in the column looks like a whole number. But each of those eight values is an identifierA code that points at one particular thing and does no other work. Because it labels rather than measures, a column of them can be sorted and matched but never sensibly totalled. rather than an amount, so their average is real arithmetic that describes nothing whatsoever. Hold that column as text and the mistake becomes impossible rather than merely unlikely.

The second is that the row order is a fact about the table rather than a fact about the market. Row 11 and row 12 both describe NB-03 in month 2, one filed on day 5 and one filed on day 19. The two rows sit next to each other because the office typed them that way, not because the market did anything twice in a row. Sort the table by takings and they move apart. Nothing about Neelbagh changed; only the positions did. Written down, the point sounds obvious, and it is still the single most common source of a wrong answer. A reader who has looked at the same file in the same order for a week starts treating position 12 as though it means something.

The third promise is the least glamorous and the most load bearing. Every column has a name, so the question is asked in the office's own vocabulary. The question does not ask for the sixth thing along. The question asks for takings_rupees, and if somebody inserts a column tomorrow the question still means what it meant yesterday.

What is actually done to a table, and how few things are there?

Four moves, and they cover almost every question an analyst is asked of a single record. Look at its shape. Take one column. Keep some rows and drop the rest. Put rows into groups and read something off each group. Everything else on a first pass is one of those four wearing different clothes.

Looking at the shape means asking the table how many rows and how many columns it has, before asking it anything interesting. The table answers 32 and 8. The shape is not a formality. A file that lost four rows on the way in would read 28 and 8, and 28 and 8 look perfectly healthy in every step that follows. The shape check is the only cheap place to catch it.

Here is the first of those moves written out. The code runs on the 32 rows printed above and nothing else.

# `rows` holds the 32 lines printed in the figure above, one entry per line.
# `columns` holds the eight names read off the header row of that same file.
shape = (len(rows), len(columns))
print(shape)                     # (32, 8)
print(shape[0] * shape[1])       # 256 cells in all

Asks the table for its shape and gets 32 rows by 8 columns, making 256 cells, before anything at all is computed from it.

Taking one column and keeping some rows are written out below, and notice what the comment is doing. The comment says why the test is there rather than restating the line. The line is already visible.

# takings_rupees is the sixth name in `columns`, so it is position 5 in every row.
takings = [r[5] for r in rows]
print(len(takings))              # 32, exactly one value per row

# a blank cell holds no number, so it cannot join anything that gets averaged
usable = [t for t in takings if t != ""]
print(len(usable))               # 31, and the one that went is NB-05 in month 2

Takes the takings column, gets one value per row, then keeps only the cells carrying a number and reports 31 out of 32.

A result computed from data the reader can see is a result the reader can check. A result computed from data fetched somewhere out of sight has to be taken on trust instead. The 32 rows above are printed in full for exactly that reason, and an argument resting on reading the count before reading the figure cannot then rest on a number nobody was shown.

How are rows put into groups, and does the record stay whole?

Grouping is the fourth move and the one worth slowing down for. Grouping is where a language feature and a finance task stop being the same thing. The mechanical description is dull: pick a column, and every row goes into the box named by its value in that column. The finance description is the whole job. The column grouped by is the question being asked, and switching the column switches the question without touching a single figure.

Group the Neelbagh returns by stall_id and the question is how each stall did. Group them by month and the question is how the market did over time. Group them by category and the question is which kinds of trade the market runs on. Same 32 rows, same takings, three different questions, and no arithmetic in between.

Here is the ladder, and it is worth reading the counts before reading anything else.

Read the right hand column first: all five groupings hold all 32 rows, so nothing was lost and only the question changed. That column staying still is the point. A grouping that returned 30 rows would be a grouping that quietly threw two away, and that is worth knowing before a single takings figure is read.

The code that produces those counts is four lines long and the last three of them are the check rather than the answer.

# every row goes into the box named by its value in one chosen column
groups = {}
for r in rows:
    groups.setdefault(r[0], []).append(r)      # r[0] is stall_id

print(len(groups))                              # 8 groups
print([len(g) for g in groups.values()])       # [4, 4, 5, 3, 4, 4, 4, 4]
print(sum(len(g) for g in groups.values()))  # 32, so nothing was lost

Groups the 32 rows by stall_id into 8 boxes holding 4, 4, 5, 3, 4, 4, 4 and 4 rows, and confirms the sizes still add back to 32.

Grouping by month instead does something different. Four groups, as four months would suggest. But the sizes are 8, 9, 7 and 8, and a market of 8 trading stalls filing one return each should give four groups of 8. Two of those four numbers are the record's own faults arriving as counts rather than as figures. The 9 is month 2, where NB-03 filed a return on day 5 and then filed a revised one on day 19, so the office typed two rows for one stall month. The 7 is month 3, where NB-04 was shut and no return exists at all.

What do the Neelbagh returns look like grouped five ways?

The shape comes first. Nothing interesting is asked of a table before its shape is known. The record is 32 rows and 8 columns, so 256 cells, and 31 of the 32 takings cells carry a number. Then the five groupings, one after another, on exactly those rows.

By stall_id: 8 groups holding 4, 4, 5, 3, 4, 4, 4 and 4 rows. By stall_name: 9 groups, and the two extras are one stall written two ways, Harit Greens on 3 rows and Harit Green on 2. By month: 4 groups holding 8, 9, 7 and 8. By category: 5 groups holding 4, 12, 4, 3 and 9, the 12 being the three cooked food stalls over four months. By filed_on_day: 5 groups again, holding 1, 16, 7, 7 and 1. Months 1 and 4 were both filed on day 6, and the two odd single rows are NB-03 filing on day 5 and then on day 19. One table, five questions, and all 32 rows present every single time.

What does the count of groups tell the analyst before any figure does?

The count of groups tells the analyst whether the column just grouped by means what it was assumed to mean. The sentence is small and carries a lot of weight, so here it is against the record. Only one of the five group counts is the number of stalls that traded, and it is the 8 from stall_id. The 4 from month is the number of months. The 5 from category is the number of trades. The 5 from filed_on_day is the number of distinct filing days. And the 9 from stall_name is not the number of anything in the market at all.

Grouping by the name gives 9 because NB-03 is written Harit Greens on its months 1 and 2 rows and Harit Green on its months 3 and 4 rows, and month 2 carries two rows. So the long spelling holds 3 rows, the short spelling holds 2, and one stall appears as two groups. The split spelling is a fault found by counting rather than by looking, and that is exactly why it survives. Nobody scanning a printed name column notices a missing letter s halfway down. A count of 9 against an expected 8 takes about a second and does not depend on anybody being sharp that morning.

Then there is the tie, and it needs labelling in the same breath as it is stated. Category gives 5 groups and filed_on_day gives 5 groups. The equality of the two counts is arithmetic on this particular record and not a finding about anything. The two columns do not group the same rows in any sense: the biggest overlap between any one category group and any one filing day group is 6 rows out of 32. Cooked food and day 6 are not versions of each other, they merely happen to produce the same number of boxes. The check that settles it takes one look: the two groupings set side by side, to see whether the same rows land together. Reading a shared count as a shared meaning invents a relationship out of a coincidence.

What is the answer, and what count sits underneath it?

The question the Neelbagh market office actually asked was what the average takings per stall month were. On the cleaned table that answer is Rs 54,196.55/- over 29 usable stall months, from a total of Rs 15,71,700/-. Both halves of that sentence travel together, always.

The conventionA rule about how something is counted, agreed in advance and put in writing. Its point is that somebody else following the note lands in exactly the same place. the market office settled says which takings cells are allowed into the average: a cell joins only if it carries a number. A blank does not. The office code standing for no return received does not join. The code is a message rather than an amount. A real zero does join. A stall that shut for a month and reported nothing genuinely took nothing. Apply that to the cleaned table and 29 cells qualify.

# `clean` is the 29 row table left after the record's known faults were corrected.
# the counting rule: a cell joins only if it carries a number. a real zero does.
figures = [r[5] for r in clean]
total = sum(figures)
print(total, len(figures))              # 1571700 29
print(round(total / len(figures), 2))   # 54196.55, and the 29 goes with it

Sums the 29 usable takings figures to 15,71,700 and divides by 29 to reach 54,196.55, printing the count beside the total so neither can travel alone. Python prints those totals without the commas an Indian reader writes by hand, so the same two numbers are Rs 15,71,700/- and Rs 54,196.55/-.

Printing the count beside the figure is not a courtesy, it is the figure being defined. The same total of Rs 15,71,700/- divided by 30 rather than 29 gives a different average, and both are correct arithmetic. The two averages differ only in the denominatorThe count sitting under an average, which decides what the average is actually an average of. Two people can divide the same total by two different counts and both be right, so the count has to be stated. used, and a reader can tell which one it was only if the count is stated. An average printed alone is a number somebody has to trust. An average printed with its count is a number somebody can check.

Which two counts does an analyst read after every single step?

The row count and the group count. Two counts, and no more. Both are cheap enough to run every time, and both are therefore the first two things dropped by anybody working at speed.

Picture a credit analyst at a lender with a borrower's monthly sales file in front of them, sent across to support a working capital limit. The file arrives with a year of rows. The analyst filters it to one product line and computes an average monthly sale. The row count says whether they still have the record they started with, and the group count says whether the column they filtered on means what they assumed it meant. If the file had 480 rows and the filter left 61 when the borrower described eleven months of trading, something is wrong and it is wrong now, not at the credit committee.

The same discipline is what a household does without naming it. The notes are counted at the bank counter before anyone leaves it, not out of distrust of the cashier, but because that is the last moment the count is cheap. Ten seconds at the counter or an argument tomorrow.

Two counts recorded at every step turn a set of results into a set of results with provenanceWhere a number started and what happened to it at each step, kept in writing. A figure somebody doubts can then be traced back to the cell it came out of.. A step whose row count was never read is a step nobody actually saw, so the trail behind that figure has a hole in it exactly there. The value is not in any single check. The value is that when a figure looks wrong three weeks later, the counts can be walked back through to find the step where the record stopped being the record, instead of starting the whole thing again.

One last thing is worth stating, and it explains why counting carries so much of the work. Every count above is read against one sentence about what a row is: one row of the Neelbagh record describes one stall in one month. The sentence naming what one row stands for is the grainOne line of this table stands for one thing, and this is the sentence naming that thing. Until it is on paper, nobody can say what a count of those lines has counted. of the table, and once it is written down, 9 groups from a column that should describe stalls is visibly wrong rather than merely surprising.

The Neelbagh market, the Neelbagh stall record, the Neelbagh day book and the market office are invented.
Educational material. Not advice on any investment, tax, budget or market position.