Handling Missing Values in Pandas

Hello students, welcome to Kwickprep. Anu was absent for the English test. Should her mark be zero? No, because zero means she wrote the test and scored nothing. Today we will see what missing data looks like, how to find it, and how to drop it or fill it. Most importantly, we will learn how to decide which one is right.

In real projects, data is almost never complete. A student may be absent, so there is no mark. A temperature sensor may stop working for an hour. A person filling an online form may skip a question. And some values get lost while copying or joining files. A machine learning model learns from data, so we must clean these gaps before training.

Here is a small marks table. From NumPy we import nan, which is the special value for missing. Pandas prints a missing value as N a N, which stands for Not a Number. Anu has no English mark, and Raj has no Science mark. Python's None is also shown as N a N. Notice the marks became decimals like eighty eight point zero, because N a N is a decimal value.

Here is a surprise. Pause and predict. Is N a N double equals N a N True or False? It is False. N a N is not equal to anything, not even to itself. So we can never find missing values with double equals. Pandas gives us special methods instead.

The first method is is null. It checks every cell and returns True where a value is missing, and False where a value is present. Anu's English cell is True, and Raj's Science cell is True. Everything else is False. Is N A is another name for the same method.

A big file can have thousands of rows, so we count instead of looking. Is null dot sum adds up the Trues in each column, because True counts as one. English has one missing value, and Science has one. This one line is usually the first step when you open any new data file.

The opposite method is not null. It returns True where a value is present. We can use it as a filter. Df of df Eng dot not null keeps only the rows where English has a mark. So Om and Raj stay, and Anu is left out.

Now we act on the gaps. Drop N A removes every row that has at least one missing value. Anu and Raj each have one gap, so both rows are removed. Only Om is left. We lost two of our three students, so dropping can throw away a lot of data.

Drop N A has a few useful options. Axis equals one drops columns instead of rows. How equals all drops a row only when every value in it is missing. Thresh equals two keeps rows that have at least two real values. And remember, drop N A gives back a new DataFrame. Your original df does not change unless you store the result, like df equals df dot drop N A.

The other choice is to fill the gaps. Fill N A replaces every N a N with the value we give. Here we filled with zero. But think about it. Anu was absent, and now she looks like she scored zero. Zero is right only when missing really means zero, like no sales on a holiday.

A fairer fill for marks is the mean, which is the average. Df dot mean finds the average of each column. English has eighty eight and sixty four, so the mean is seventy six. Science has ninety one and sixty four, so the mean is seventy seven point five. Each gap is filled with its own column's mean.

Pause and predict. This Series has four, a missing value, and seven. Is the mean eleven divided by three? No. Pandas skips N a N, so count is two. The mean is eleven divided by two, which is five point five. That is why the mean fill in the last slide used only the real marks.

So how do we decide? If only a few rows have gaps and the data is big, drop those rows. If a column is mostly empty, it is not useful, so drop the column. For numbers, fill with the mean, or with the median when there are extreme values. For categories like city, fill with the most common value, called the mode. Fill with zero only when missing truly means zero.

Here is the same decision as a flowchart. First, count the gaps with is null dot sum. Is a column mostly empty? If yes, drop that column. If no, ask, are only a few rows affected? If yes, drop those rows. Otherwise, fill the gaps with the mean, the most common value, or zero, whichever makes sense.

Let us revise what we learned today. Missing values show as N a N, which means Not a Number. Is null finds the gaps, and is null dot sum counts them. Not null finds the real values. Drop N A removes the gaps, and fill N A fills them. And always decide by how much data is missing and what the gap really means. Try this on any data file and count its gaps first.