Mastering Pandas DataFrame fillna() for Handling Missing Data

Introduction to fillna()

The fillna() function is used to replace NaN or null values in a DataFrame with a specific value, or a method-based imputation. The function signature is as follows:

codeDataFrame.fillna(value=None, method=None, axis=None, inplace=False, limit=None) 

• value : Scalar, dict, Series, or DataFrame. The value to use to fill missing values.
• method : {'backfill', 'bfill', 'pad', 'ffill', None}, default None. Method to use for filling holes in reindexed Series.
• axis : {0 or 'index', 1 or 'columns'}, default None.
• inplace : Boolean, default False. If True, fill in-place.
• limit : Int, default None. If method is specified, this is the maximum number of consecutive NaN values to forward/backward fill.

Filling NaN Values with a Specific Value

You can replace all NaN values in a DataFrame with a specific value:

codeimport pandas as pd 
import numpy as np 

# Sample DataFrame with NaN values 
data = {'Name': ['John', 'Anna', np.nan, 'Linda'], 
    'Age': [28, np.nan, 34, 29], 
    'City': ['New York', 'Paris', 'Berlin', np.nan]} 
    
df = pd.DataFrame(data) 

# Replace all NaN values with a specific value 
df.fillna('Unknown', inplace=True) 
print(df) 

Using a Dictionary to Replace NaN

You can use a dictionary to replace NaN values with different values for each column:

code# Replace NaN values with different values for each column 
df.fillna({'Name': 'No Name', 'Age': 0, 'City': 'No City'}, inplace=True) 
print(df) 

Forward Filling and Backward Filling

You can fill NaN values using the forward fill or backward fill method:

code# Forward fill 
df.fillna(method='ffill', inplace=True) 

# Backward fill 
df.fillna(method='bfill', inplace=True) 

Limiting the Number of NaN Values to Fill

You can limit the number of consecutive NaN values to fill:

code# Limit the number of NaN values to fill 
df.fillna(0, limit=1, inplace=True) 

Conclusion

Handling missing data is crucial in ensuring the accuracy and reliability of your analysis. The fillna() function in Pandas provides a versatile and powerful tool to address this issue, offering various methods to replace or impute missing values. Whether you’re dealing with a small dataset or large, mastering the use of fillna() is essential for any data scientist or analyst.