Missing Data
14 min
Real data has gaps. pandas represents a missing value as NaN
(Not a Number) — writing None into a numeric column gets silently
converted to NaN. .isna() / .notna() check for it (NaN famously isn't
even equal to itself, so x == None doesn't work reliably for detection).
import pandas as pd
import numpy as np
s = pd.Series([1, None, 3])
print(s.isna()) # False, True, False
print(s.isna().sum()) # 1 -- count of missing values
Filling with the column's own mean (rather than a fixed value like 0) is a common default when a missing value shouldn't drag down an average.
A genuinely useful gotcha: a numeric column containing ANY None/NaN
gets upcast to float64, even for values that were originally whole
numbers — because NaN only exists as a float, not an int.
pd.Series([1, None, 3]).dtype # float64 -- not int64, even though 1 and 3 are whole numbers
pd.Series([1, 2, 3]).dtype # int64 -- no NaN, stays int
That's why .fillna(0) on a column that had any missing values often
gives you 0.0 back, not 0 — the whole column already became float the
moment a NaN entered it, before .fillna ever ran.
Write `fill_missing(data, col, value)`: build a DataFrame from `data`, and return column `col` with every missing value replaced by `value`, as a list (`.fillna(value).tolist()`).
Write `drop_missing_rows(data)`: build a DataFrame from `data`, drop every row that has at least one missing value (`.dropna()`), and return the number of rows remaining.
What does pandas use to represent a missing numeric value?
You can fill or drop missing values, and know why a column with any NaN in it becomes float64 even if every visible value looks like an integer.