A Comprehensive Guide to NumPy File IO

Introduction

link to this section

Data analysis and scientific computing often involve dealing with large datasets that need to be stored efficiently and retrieved when required. NumPy, a foundational package for numerical computing in Python, offers a variety of functions that enable users to save and load data to and from files with ease. This guide will walk you through how to read and write arrays to files using NumPy.

Datathreads Advertisement - On-Premise ETL,BI, and AI Platform

Writing NumPy Arrays to Files

link to this section

NumPy provides several functions to save arrays to files in various formats. The most common formats are binary (.npy, .npz) for storage efficiency and text files for readability.

Saving to Binary Files

np.save()

To save a single array to a binary file with a .npy extension: 

import numpy as np
#Create a random NumPy array 
array_to_save = np.random.rand(5, 5)

#Save to a .npy binary file 
np.save('my_array.npy', array_to_save)

The .npy format is a standard binary file format in NumPy for persisting a single arbitrary NumPy array on disk.

np.savez() and np.savez_compressed()

For saving multiple arrays in one file, you can use np.savez() or np.savez_compressed() for uncompressed and compressed files, respectively. 

# Create multiple NumPy arrays 
array_one = np.arange(10) 
array_two = np.arange(10, 20)

#Save multiple arrays to a .npz file 
np.savez('my_arrays.npz', array_one=array_one, array_two=array_two)


#For compressed files 
np.savez_compressed('my_arrays_compressed.npz', array_one=array_one, array_two=array_two)

Saving to Text Files

np.savetxt()

For saving an array to a text file: 

# Create a NumPy array 
array_to_save = np.arange(12).reshape(4, 3)

#Save to a text file 
np.savetxt('my_array.txt', array_to_save)

By default, np.savetxt() saves data in scientific notation. You can change the format using the fmt parameter.

Datathreads Advertisement - On-Premise ETL,BI, and AI Platform

Reading NumPy Arrays from Files

link to this section

Reading arrays from files is as straightforward as writing them.

Loading from Binary Files

np.load()

For loading .npy or .npz files: 

# Load a .npy file 
loaded_array = np.load('my_array.npy')

#Load a .npz file 
loaded_arrays = np.load('my_arrays.npz') 
array_one = loaded_arrays['array_one'] 
array_two = loaded_arrays['array_two']

Loading from Text Files

np.loadtxt()

For loading an array from a text file: 

# Load from a text file 
loaded_array_from_text = np.loadtxt('my_array.txt')

Best Practices for File IO with NumPy

link to this section
  1. Binary vs. Text : Use binary formats for efficiency and text formats for human-readable files.
  2. Compressed Files : Use np.savez_compressed() to save disk space when dealing with large datasets.
  3. Memory Mapping : Use np.memmap for accessing small segments of large files on disk, without reading the whole file into memory.
  4. File Extension : Always use the .npy or .npz extension for binary files to avoid confusion and ensure compatibility.
Datathreads Advertisement - On-Premise ETL,BI, and AI Platform

Conclusion

link to this section

NumPy's file IO capabilities simplify the process of saving and loading data, making it an invaluable tool for anyone working with datasets in Python. By understanding how to use these functions effectively, you can integrate data persistence into your scientific computing workflow efficiently.