A one-day tick file has 2% of rows at price zero, 5% duplicated timestamps and one print of 8,120 between prints of 812. Identify each error, fix it, and show how each would distort daily realised volatility.

Clean before you compute, and log every change. The zeros are missing prints, not prices: drop them, because a log of zero or a return of minus 100% makes the statistic undefined. The duplicated timestamps are a feed replay: keep one row per minute, and they would otherwise understate volatility by about 2%. The 8,120 is a decimal shift that reverses the next minute: correct or drop it, because left in it lifts the day's realised volatility from 1.55% to about 326%, 210 times the truth.

Name the cause, because the cause tells you the fix. A price of exactly 0 is never a trade; it is a field the feed filled with its default when no print arrived, or a parser reading a blank as a number. Treat it as missing. A repeated timestamp is usually a replayed packet or two feeds merged without de-duplication; check whether the repeated rows carry the same price, because identical rows are harmless to drop while different prices at the same stamp mean the file's ordering cannot be trusted. A print of exactly ten times the price that reverses a minute later is a decimal shift, a dropped or added digit somewhere between the exchange and your file, and the factor of exactly 10 with a full reversal is the signature that separates it from a real move. A real jump that does not reverse, say a stock going from 812 to 406 and staying there, could be a split or news, and that one you investigate rather than delete.

Work the clean number first so there is something to compare with. Minute returns with a standard deviation of 0.08% over 375 minutes give a daily realised volatility of 0.08% times the square root of 375, about 1.55%. The zeros do not distort the statistic; they destroy it: a log return into a zero price is the log of zero, and a simple return is minus 100% followed by a division by zero on the way out. Code that silently drops the resulting NaN values will also drop the two returns around each zero and report a number that looks fine, which is worse than an error message.

The duplicates are the quiet fault. If the repeated rows carry the same price, each adds a zero return, so the sum of squared returns is unchanged but there are 394 returns instead of 375; a volatility computed as the standard deviation of minute returns times the square root of 375 falls to about 1.51%, an understatement of about 2%. Small, but it is a bias that appears every day in the same direction, and anything built on the row count, such as volume per minute, is 5% wrong.

The decimal shift is the loud fault: left in, the day's realised volatility is about 326% instead of 1.55%, 210 times too high, and with simple returns instead of log returns the figure is about 904%. A single row has done that. A spreadsheet that says one of your cells is a thousand times the others does not change the average much if you are summing values, but a volatility squares the differences, so one outlier is not diluted by 374 good minutes; it replaces them.

Fix in the order that keeps each step checkable. Drop the zero prices and count them. Sort by timestamp, drop exact duplicate rows, and if two rows share a stamp with different prices keep the last and flag the minute. Then run a plausibility filter on returns: any minute return beyond a threshold, say ten clean standard deviations, that reverses fully within one or two prints and matches a factor of 10 or 100 is a decimal shift, corrected by dividing or dropped, and logged. Never fix silently: write the counts of rows dropped and corrected next to the result, and assert at the end that prices are positive, timestamps unique and increasing, and no return exceeds the threshold, so the next file that breaks one of those stops the pipeline instead of poisoning the statistic. Say the limitation too: a filter that deletes every large return will also delete the genuine crash, so the rule needs the reversal test, not just the size test.