Mastering Data Deduplication in Pandas: Using drop_duplicates()

Understanding drop_duplicates()

The drop_duplicates() function returns a DataFrame with duplicate rows removed, considering all columns.

codeDataFrame.drop_duplicates(subset=None, keep='first', inplace=False, ignore_index=False) 

• subset : column label or sequence of labels, optional. Only considers certain columns for identifying duplicates. By default, all columns are used.
• keep : {‘first’, ‘last’, False}, default ‘first’. Determines which duplicates (if any) to keep.
  • ‘first’: Drop duplicates except for the first occurrence.
  • ‘last’: Drop duplicates except for the last occurrence.
  • False: Drop all duplicates.
• inplace : bool, default False. Whether to drop duplicates in place or to return a copy.
• ignore_index : bool, default False. If True, the resulting axis will be labeled 0, 1, …, n - 1.

Removing All Duplicate Rows

To remove all duplicate rows from your DataFrame, simply call the drop_duplicates() method.

codeimport pandas as pd 

# Sample DataFrame 
data = {'Name': ['John', 'Anna', 'Peter', 'Linda', 'John'], 
    'Age': [28, 24, 34, 29, 28], 
    'Salary': [70000, 80000, 120000, 110000, 70000]} 
    
df = pd.DataFrame(data) 

# Removing all duplicate rows 
df_unique = df.drop_duplicates() 
print(df_unique) 

Keeping the Last Occurrence

If you want to keep the last occurrence of the duplicate row and remove the rest, you can use the keep='last' parameter.

code# Keeping the last occurrence of the duplicate rows 
df_unique = df.drop_duplicates(keep='last') 
print(df_unique) 

Removing All Occurrences of Duplicates

If you want to remove all occurrences of duplicate rows, you can use keep=False .

code# Removing all occurrences of duplicate rows 
df_unique = df.drop_duplicates(keep=False) 
print(df_unique) 

Specifying Columns to Identify Duplicates

You might want to identify duplicates based on specific columns. You can achieve this by passing the column names to the subset parameter.

code# Removing duplicates based on specific columns 
df_unique = df.drop_duplicates(subset=['Name', 'Age']) 
print(df_unique) 

In-Place Removal

If you wish to remove the duplicates directly in the original DataFrame, you can set the inplace parameter to True.

code# Removing duplicates in-place 
df.drop_duplicates(inplace=True) 
print(df) 

Resetting Index After Dropping Duplicates

When you drop duplicates, the index of the DataFrame might get messed up. You can reset it using the reset_index() method, combined with ignore_index=True .

code# Resetting index after dropping duplicates 
df_unique = df.drop_duplicates().reset_index(drop=True) 
print(df_unique) 

Conclusion

Removing duplicate data is a crucial step in the data cleaning process. With Pandas’ drop_duplicates() function, you can easily identify and remove duplicate rows from your DataFrame, ensuring that your data analysis is based on accurate and reliable data. By mastering the use of this function, you can significantly enhance the quality of your data and the reliability of your analysis results. Happy data cleaning!