Records, Fields and the Honest Clean
🎯 Today's mission briefing
We are learning to design a dataset that can actually answer our question — records, fields and data types — then collect or source real data, clean it honestly, and structure it into related tables so it cannot contradict itself.
You'll know you've got it when:
- I can turn a real question into a dataset design, stating what one record represents and what fields it needs
- I can choose a sensible data type and unit for every field, and say why the type matters
- I can source or collect real data and document exactly where it came from and how
- I can find and either fix or honestly flag missing values, inconsistent formats, duplicates and impossible outliers
- I can split one repeating table into two related tables joined by a key, and explain what problem that solves
- I can state in one sentence what my cleaned dataset can answer and what it cannot
Three hundred rows and not one usable answer
Connects to what your guest already knows and makes them curious. Activating prior knowledge is one of the strongest predictors of new learning.
The canteen at a Queensland footy club has been keeping records for two seasons. Three hundred and forty rows, handwritten in an exercise book, then typed up by four different volunteers. It should be able to answer a simple question — should we order more sausages or more chips for a Saturday home game? — and it cannot answer it at all. Dates are written as 3/6, 6 March, 06-03-24 and 'Sat'. One volunteer wrote sales as dollars, another wrote them as number of items. There are two rows for the same game because someone typed it up twice. A row reads 'sausages, 400' for a game where sixty people turned up. Nothing here is dishonest. Everyone tried. But the dataset was never DESIGNED, and a dataset that was never designed answers nothing, no matter how many rows it has. This lesson is about being the person who designs it first.